1. Field of the Invention
This invention is directed to a method and apparatus for implementing a neural network. More particularly, it is directed to implementing a state-analog neural network in which analog computations are made for each of the network nodes based upon frozen values of the analog signals from the previous computation step.
2. Background Information
Neural networks, or nets, are, for the purposes of this discussion, devices which perform computations or analysis using multiple summing nodes with thresholds (neurons), each of which receives inputs from fixed or variable gain amplifiers (synapses). Each synapse receives input from a neuron or from an external signal source. The most general neural networks are those in which every neuron receives input via a synapse from every other neuron (including itself) and every external input. In such a network, the number of synapses is equal to the square of the number of neurons plus the product of the numbers of neurons and inputs. Neural networks are "programmed" by specifying the gains (strengths) of the synapses. In most applications, a subset of the synapses in the most general network can be omitted because their strengths are insignificant.
In some applications, the strengths of synapses may be modified on the basis of how closely the outputs of neurons approximate the desired outputs, given certain inputs. In other words, modification is based upon the errors at the outputs. If this is done under external control, it is referred to as "training". If it is done using circuitry internal to the neural net, it is referred to as "self-organization". Both are cases of "learning".
Neural nets may be organized in "layers", where each layer consists of circuitry engaged in a coordinated parallel processing of inputs from a previous layer. When more than one layer exists, it may be necessary to use some form of backward propagation of errors from the output layer for synaptic strength modifications in the earlier layers.
Neural networks operate to some extent in a manner similar to portions of the nervous systems of animals. Like such physiological systems they involve multiple parallel processing paths. Memory is distributed rather than concentrated in specific locations as in a digital computer. Such distributed circuits can act as optimal filters and extract information from background noise. As such, they are also useful in making rapid, best guess approximations. They have application in recognizing complex signal patterns such as images which could be used for instance in security systems for identifying faces or finger prints. They can also be used for knowledge representation and manipulation, providing a basis for cognitive functions and intuition in artificial intelligence. They can also provide coordinated output of complex signal patterns for control, e.g., for coordinated movements of limbs in robotic walking.
Implementation of neural networks at this point has mostly been achieved with simulation on general purpose digital computers. Now several groups are working on implementation in hardware. To the best of my knowledge, hardware implementations have been of the following types: (1) digital, i.e., logic which is discrete-voltage, largely discrete-time; and (2) analog, which is continuous-time, continuous-voltage.
Digital systems, such as computers, utilize state logic. State logic is discrete-time, discrete-voltage logic which in its purest form has the following constraint: all signals which are inputs for computation are acquired by computing elements at specific times, determined by a clock. The time of acquisition is a time when all the signals are known to be the final values obtained in the previous compute cycle, and before the initiation of the next compute cycle. This generally means that the outputs of computing elements are captured at clock transitions, and held stable so that the computing elements can perform their next computations on stable values. This is necessary because there often exist feedback or feed-forward paths which otherwise would cause inputs to a computing element to change while it is performing its computation. In parallel processing applications such as neural nets, it is quite common for the outputs of computing elements to influence each other and some method of "freezing" the results of one computation step is needed so that the next computation step can be performed on a stable input value. Also, different parallel computing elements can take different times to respond to transients, either because they have different slew rates or because they have to slew different amounts. This can lead to devices which should operate in synchrony becoming out of phase. Finally, if it is necessary for a circuit to perform a sequence of operations, especially if the sequence may vary from one time to another, it is easier to control the sequencing with a clocked system.
For each state of a state logic device there is customarily a unique timing pulse which causes the execution of computations (or storage of results of computations) required in that state. For fool-proof operation, it is often necessary for these pulses (and their states) to be separated in time, so that one state is over before the next begins. Finally, in many state logic devices it is necessary to use master/slave storage, where two timing pulses (or opposite transitions of one pulse) are used to strobe master and slave storage elements. In this way, a computing element can never see its own output as one of the inputs during clocking. Instead, each computing element output provides input to a master storage device, and each computing element input comes from the output of a slave storage device. Another function of the timing pulses in state logic is to control steering logic to route signals.
State logic implementation of a neural network with a meaningful number of nodes through simulation by a general purpose digital computer has not produced real-time solutions due to the large number of parallel paths and the interactions generated by even a moderate number of neurons. Parallel processors, each performing calculations for several neurons, would help, but would require complex interconnections between processors.
Pure digital implementation with dedicated processors for each neuron and synapse, even through large scale integration on a single chip, would require considerable real estate. The large number of interconnections required in any neural network is compounded in such a system where, for instance, eight bit signals would be needed for reasonable accuracy.
Analog circuits occupy less real estate on a chip and require only a single lead for transmitting each analog signal. However, differences in slew times in individual circuits in a massive array of interconnected analog circuits can produce race conditions which, combined with feedback within a network, can effect system stability. In addition, it is customary in analog circuits to store analog signals where required in sample and hold circuits. Such circuits, in which the analog signal is stored as a charge on a capacitor, are not suitable for prolonged storage, as the stored charge decays with time. In neural networks, some analog signals must be stored for very long periods of time. For instance, synaptic strength signals may remain unchanged for indefinite periods of time, such as when the net is not operating in a learning mode or when a particular synaptic strength is not affected in a certain learning situation.
There is a need therefore for a neural network having a meaningful number of nodes which can operate reliably in real-time while occupying a minimum of physical space.
It is therefore a primary object of the invention to provide a method and apparatus for implementing such a neural network.
It is another important object of the invention to provide such a method and apparatus which function in a state analog mode.